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The  purpose  of  this  study  was  to  determine  the  efficacy  of  providing  target  location  information 
via  head-coupled  visual  and  spatial  audio  displays  presented  in  adaptive  and  non-adaptive  configura¬ 
tions.  Twelve  USAF  pilots  performed  a  simulated  flight  task  in  which  they  were  instructed  to  maintain 
flight  parameters  while  searching  for  ground  and  air  targets.  The  integration  of  visual  displays  with 
spatial  audio  cueing  enhanced  performance  efficiency,  especially  when  targets  were  most  difficult  to 
detect.  Several  of  the  interface  conditions  were  also  associated  with  lower  ratings  of  perceived  mental 
workload.  The  benefits  associated  with  multi-sensory  cueing  were  equivalent  in  both  adaptive  and  non- 
adaptive  configurations. 


INTRODUCTION 

An  important  consideration  for  implementing  displays  is 
their  interaction  with  existing  interfaces.  Cockpits  have  lim¬ 
ited  space  for  information  display,  and  there  is  already  a  high 
visual  demand  that  may  contribute  to  information  overload 
and  increase  the  probability  of  pilot  error  (Reising,  Liggett,  & 
Munns,  1999;  Weinstein  &  Wickens  1992).  A  promising  ap¬ 
proach  to  counteracting  visual  overload  is  the  use  of  multi- 
sensory  adaptive  interfaces,  in  which  automation  is  employed 
to  control  the  delivery  of  information  to  pilots  so  that  they 
receive  the  right  information  in  the  right  format  at  the  right 
time,  and  are  not  otherwise  exposed  to  that  information  (Het¬ 
tinger,  Cress,  Brickman,  &  Haas,  1996;  Hollnagel,  1988). 
Along  this  line,  the  benefits  associated  with  the  adaptive  inter¬ 
face  approach  may  be  best  realized  in  extreme  environments, 
where  task  demands  often  exceed  the  perceptual  and  cognitive 
capabilities  of  the  operator. 

For  example,  tactical  air  missions  place  a  high  visual  load 
on  pilots,  who  must  rapidly  acquire  and  synthesize  informa¬ 
tion  from  displays  inside  the  cockpit  and  events  outside  of  the 
cockpit  (Martin-Emerson  &  Wickens,  1997).  While  the  de¬ 
velopment  of  head-up  (HUD)  and  helmet-mounted  displays 
(HMD)  has  been  shown  to  improve  performance  in  target  de¬ 
tection  scenarios  (Fadden,  Ververs,  &  Wickens,  1998;  Os¬ 
good,  Wells,  &  Meador,  1995),  these  technologies  have  sev¬ 
eral  limitations  which  may  constrain  their  effectiveness  (e.g., 
narrow  FOV,  excessive  helmet  weight,  etc.). 

One  potential  solution  for  offsetting  these  limitations  is 
the  use  of  spatial  audio  displays,  which  have  been  shown  to 
enhance  target  detection  performance  and  reduce  overall  lev¬ 
els  of  workload  during  visual  search  tasks  (Bolia,  D’Angelo, 
&  McKinley,  in  press;  Nelson,  Hettinger,  Cunningham, 
Brickman,  Haas,  &  McKinley,  1998;  Perrott,  Cisneros, 
McKinley,  &  D’Angelo,  1996).  Despite  these  recent  empirical 
findings,  similar  investigations  conducted  in  operationally- 
relevant  environments,  such  as  tactical  aviation,  have  been 
sparse  (Bronkhorst,  Veltman,  &  van  Breda,  1996).  The  pur¬ 
pose  of  the  present  investigation  was  to  evaluate  the  role  of 


adaptive  integration  of  visual  and  spatial  audio  displays  for 
target  detection  and  designation  during  a  simulated  low-level 
terrain  avoidance  flight  task. 

METHOD 

Participants 

Twelve  pilots  (11  males  and  1  female)  serving  at  Wright- 
Patterson  Air  Force  Base  participated  in  the  study.  They 
ranged  in  age  from  32  to  51  years,  with  a  mean  of  40  years. 
All  reported  normal  or  corrected-to-normal  vision,  and  normal 
hearing.  Each  pilot  had  at  least  1500  hours  of  flight  experi¬ 
ence  in  military  aircraft,  with  a  mean  of  2652  hours  logged. 

Experimental  Design 

Seven  interface  conditions  were  combined  factorially 
with  two  target  types  (ground  and  air)  and  two  initial  target 
location  conditions  (within  and  beyond  ±  15°  of  the  partici¬ 
pant’s  instantaneous  head  orientation)  in  a  completely  within- 
subjects  design.  The  interface  conditions  comprised  a  non¬ 
cueing  control  and  six  target-cueing  interfaces,  including  spa¬ 
tial  auditory,  unimodal  visual,  and  multimodal  (auditory  and 
visual)  displays  presented  in  fixed  or  adaptive  configurations. 

Under  all  but  the  Non-Cueing  condition,  visual  and/or 
spatial  audio  cues  were  provided  to  aid  in  locating  targets.  In 
situations  in  which  visual  cueing  was  provided,  target  location 
and  range  were  provided  by  a  look-to  line  display  (see  Figure 
1)  projected  onto  the  wide  field  of  view  dome  display  and 
slaved  to  the  pilot’s  head  position. 

Spatial  audio  cueing  consisted  of  broadband  noise  pulses 
(250  ms,  70dBA),  digitally-filtered  using  a  Convolvotron 
Audio  Rendering  System  configured  with  non-individualized 
head-related  transfer  functions,  and  presented  binaurally  over 
Sennheiser  HD250  II  headphones.  Target  range  was  indicated 
by  inter-pulse  interval  (IPI),  such  that  shorter  IPIs  indicated 
closer  targets. 
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Figure  1.  Head-slaved  visual  display  symbology.  Target  location  was  indi¬ 
cated  by  the  direction  of  an  arrow,  which  diminished  in  length  as  the  angle 
separation  between  the  pilot’s  head  orientation  and  target  location  decreased. 
Changes  in  target  range  were  portrayed  by  changes  in  the  circumference  of  a 
circle  centered  in  the  pilot’s  field  of  view. 

In  the  non-adaptive  display  conditions  (Auditory,  Visual, 
and  Auditory+Visual),  spatial  audio  and/or  visual  cues  ap¬ 
peared  whenever  a  target  was  present  in  the  field  of  regard. 
Conversely,  in  the  adaptive  display  conditions,  modality  of 
presentation  was  determined  by  the  location  of  the  target  in 
relation  to  the  orientation  of  the  pilot’s  head.  For  example,  in 
the  Adaptive  Auditory  +  Adaptive  Visual  display  condition, 
the  locations  of  targets  more  than  +15°  from  the  center  of  the 
pilot’s  head  orientation  were  cued  with  the  spatial  audio  dis¬ 
play.  However,  as  the  target  moved  within  +15°  of  the  pilot’s 
field  of  view,  the  spatial  audio  cue  was  replaced  by  the  look-to 
line  display.  In  the  Adaptive  Auditory  +  Non-adaptive  Visual 
display  condition,  the  use  of  spatial  audio  cueing  was  once 
again  determined  by  target  locations  in  excess  of  ±15°  from 
the  center  of  the  pilot’s  head  orientation,  while  the  visual 
cueing  display  was  always  present.  In  the  Adaptive  Visual  + 
Non-adaptive  Auditory  condition,  the  look-to  visual  display 
was  only  presented  when  targets  were  located  within  +15°  of 
the  center  of  the  pilot’s  head  orientation,  while  spatial  audio 
cues  were  provided  regardless  of  target  location. 

Apparatus  and  Procedure 

SIRE  Facility.  The  study  was  conducted  at  the  Air  Force 
Research  Laboratory's  Synthesized  Immersion  Research  Envi¬ 
ronment  (SIRE)  at  Wright-Patterson  Air  Force  Base,  Ohio. 
The  SIRE  facility  houses  a  fixed-base  cockpit  situated  in  the 
center  of  a  6.1  m  radius  dome  that  includes  a  high-resolution, 
large  field-of-view  (70°  vertical  by  150°  horizontal)  interac¬ 
tive  visual  display.  The  cockpit  contained  head-up  and  head- 
down  visual  displays,  throttle  and  sidestick  aircraft  controls,  a 
head  position  and  orientation  tracker,  and  a  Convolvotron 
spatial  audio  rendering  system.  Out-the-window  imagery  con¬ 
sisted  of  desert-like  mountainous  terrain.  All  aspects  of  the 
experiment  were  controlled  by  digital  computers. 

Flight  Task.  Pilots  were  instructed  to  maintain  an  air¬ 
speed  of  500  knots  while  flying  at  or  below  300"  above  ground 
level  (AGL).  The  flight  task  also  required  pilots  to  minimize 
lateral  deviation  from  a  waypoint-guided  flight  path,  com¬ 
prising  three  waypoints,  each  of  which  were  separated  by  ap¬ 
proximately  31.38  km.  Deviations  from  the  flight  path  were 


indicated  on  the  Horizontal-Situation-Display  (HSD).  Addi¬ 
tional  avionics  information,  including  aircraft  heading,  air¬ 
speed,  and  AGL  altitude  were  presented  on  a  30°  FOV  HUD. 

Target  Designation  Task.  While  maintaining  the  directed 
flight  parameters,  pilots  visually  searched  the  out-the-window 
scene  for  ground  and  air  targets,  SCUD  launchers  and  MH-53 
helicopters,  respectively.  Eight  ground  and  eight  air  targets 
were  randomly  distributed  along  the  flight  path  on  each  ex¬ 
perimental  trial.  Pilots  were  instructed  to  visually  locate  and 
designate  targets  as  quickly  as  possible  by  moving  a  head- 
coupled  aiming  reticle,  projected  onto  the  dome,  over  a  target 
and  depressing  the  appropriate  control-stick  button.  A  correct 
designation  (hit)  was  defined  as  an  appropriate  button  press 
while  the  aiming  reticle  was  within  3°  of  the  target. 

Procedure.  For  each  interface  condition,  pilots  completed 
a  block  of  five  consecutive  trials.  Immediately  following  the 
completion  of  each  condition,  pilots  assessed  the  perceived 
mental  workload  of  the  task  via  the  NASA-Task  Load  Index 
(TLX;  Hart  &  Staveland,  1988).  Search  performance  under  the 
adaptive  and  non-adaptive  interface  configurations  was  as¬ 
sessed  in  terms  of  target  designation  performance,  designation 
time,  and  patterns  of  head  motion. 

RESULTS 


Designation  Accuracy 

Mean  percentages  of  correct  detections  were  calculated 
for  all  experimental  conditions,  converted  to  arsines  (Winer, 
Brown,  &  Michels,  1991),  and  submitted  to  a  2  (initial  target 
location)  x  2  (target  type)  x  7  (interface  condition)  repeated 
measures  ANOVA.  The  results  of  the  analysis  revealed  sig¬ 
nificant  main  effects  for  target  type ,  F(l,ll)  =  463.60,  p  <  .05 
and  interface  condition ,  F(3,40)  =  61.64,  p  <  .05,  and  a  sig¬ 
nificant  target  type  x  interface  condition  interaction,  F(3,40)  = 
17.20,  p  <  .05.  These  effects  are  illustrated  in  Figure  2. 
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Figure  2.  Mean  percent  correct  designations  for  ground  and  air  targets  under 
all  interface  conditions  (NC  =  Non-Cueing;  A  =  Auditory;  V  =  Visual;  A+V  = 
Auditory+Visual).  Designation  scores  associated  with  the  non-adaptive  and 
adaptive  interface  conditions  appear  on  the  left  and  right  sides  of  the  graph, 
respectively.  The  adaptive  components  of  the  interfaces  are  indicated  by 
arrows. 


Designation  Time 

As  was  seen  in  Figure  2,  detection  performance  in  the 


Non-cueing  and  Auditory  conditions  was  significantly  worse 
than  in  all  other  interface  conditions.  These  data  were  thus 
excluded  from  the  analysis  of  designation  time  -  the  time  re¬ 
quired  to  correctly  designate  a  target  -  to  guard  against  unreli¬ 
able  estimates  of  the  means  due  to  biases  associated  with  dif¬ 
ferences  in  sample  size. 

Mean  designation  time  scores  were  submitted  to  a  2  (tar¬ 
get  type)  x  2  (initial  target  location)  x  5  (interface  condition) 
repeated  measures  ANOVA,  which  indicated  that  all  main 
effects  and  interactions  were  statistically  significant  (p<.05) 
The  target  type  x  initial  target  location  x  interface  condition 
interaction  is  illustrated  in  Figure  3. 

The  three-way  interaction  can  be  explained  by  noting  the 
significant  initial  target  location  x  interface  condition  inter¬ 
action  for  ground  targets,  depicted  in  the  upper  portion  of  Fig¬ 
ure  3,  and  the  lack  thereof  for  air  targets  (lower  portion  of 
Figure  3). 
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Figure  3.  Mean  designation  times  for  ground  (upper  portion)  and  air  (lower 
portion)  targets  as  a  function  of  initial  target  location  and  interface  condition 
(A  =  Auditory;  V  =  Visual;  A+V  =  Auditory+ Visual).  As  in  the  previous 
figure,  non-adaptive  and  adaptive  interface  conditions  appear  on  the  left  and 
right  sides,  respectively. 


Two  additional  features  that  emerge  from  the  data  pre¬ 
sented  in  Figure  3  are:  1)  the  significantly  lower  designation 
times  associated  with  the  air  targets  (X  =  3.7  s)  as  compared 
with  those  for  ground  targets  ( X  =  5.1  s);  and  2)  in  the  case  of 
ground  targets  which  initially  were  located  outside  of  the  op¬ 
erator’s  field  of  view,  the  average  designation  times  of  all 
combinations  of  multisensory  interfaces  was  825  ms  faster 
than  that  for  the  visual  interface  condition. 


Subjective  Workload 

Mean  overall  workload  ratings  on  the  NASA-TLX  are 
presented  for  the  seven  interface  conditions  in  Figure  4.  It  can 
be  seen  in  the  figure  that  ratings  for  the  Non-Cueing  and 
Auditory  cueing  conditions  fell  within  the  upper  half  of  the 
workload  scale,  and  were  approximately  50%  greater  than 
those  for  the  remaining  interfaces,  which  fell  within  the  lower 
half  of  the  scale.  A  repeated  measures  ANOVA  confirmed 
that  there  were  statistically  significant  differences  in  the 
workload  associated  with  the  several  interfaces,  F(3,30)  = 
12.80,  p  <  .05.  Post-hoc  tests  indicated  that  the  overall 
workload  associated  with  both  the  Non-Cueing  and  Auditory 
interfaces  was  greater  than  that  for  each  of  the  other  interfaces. 
The  workload  ratings  for  both  the  Non-Cueing  and  Auditory 
conditions  did  not  differ  significantly  from  each  other,  and 
none  of  the  remaining  comparisons  among  the  interface 
conditions  reached  significance  (p  >  .05). 
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Figure  4.  Mean  overall  workload  ratings  (NASA  Task  Load  Index)  for  the 
seven  interface  conditions  (NC  =  Non-Cueing;  A  =  Auditory;  V  =  Visual; 
A+V  =  Auditory+ Visual). 


Analysis  of  Head  Motion  Patterns 


Head  motion  activity  was  examined  by  plotting  head 
orientation  in  azimuth  and  elevation  as  a  function  of  time  for 
each  experimental  trial.  Inspection  of  the  resulting  figures 
revealed  two  distinct  patterns  of  head  motion  activity  -  one 
resembling  a  serial,  non-terminating  search  of  the  field  of 
regard,  the  other  indicative  of  a  series  of  brief  ballistic  head 
motions  directed  toward  the  targets.  Examples  of  these  two 
head  motions  patterns  are  illustrated  in  Figure  5.  In  general, 
the  pattern  displayed  on  the  left  side  of  the  figure  occurred  in 
conjunction  with  the  Non-Cueing  condition,  whereas  the  right 
portion  of  the  figure  characterized  head  motion  patterns  for  all 
other  experimental  conditions. 


Figure  5.  Examples  of  head  motion  patterns  associated  with  the  Non-Cueing 
(left  side)  and  Visual+Auditory  (right  side)  interface  conditions. 


The  compelling  nature  of  these  data  suggested  a 
quantitative  analysis  based  on  the  total  angular  displacement 
of  the  head.  Due  to  methodological  constraints,  it  was  not 
possible  to  analyze  these  data  in  terms  of  target  type  and 
initial  target  location.  Consequently,  mean  total  angular 
displacements  for  the  seven  interface  conditions  were 
submitted  to  a  one-way  repeated  measures  ANOVA.  Results 
of  the  analysis  indicated  a  significant  main  effect,  F( 2,20)  = 
187.03,  p  <  .05,  which  is  illustrated  in  Figure  6. 


Figure  6.  Mean  total  angular  head  displacement  under  all  interface 
conditions. 


by  the  multi-sensory  interfaces  did  not  come  at  the  expense  of 
flight  performance  (due  to  space  limitations  flight  perform¬ 
ance  data  were  not  presented  in  the  results  section).  This  out¬ 
come  suggests  that  the  multi-sensory  displays  did  not  incur  an 
additional  performance-resource  debt  (Weinstein  &  Wickens, 
1992). 

The  results  of  the  present  investigation  revealed  no  ad¬ 
vantage  for  multi-sensory  target  information  presented  in  an 
adaptive  configuration,  as  opposed  to  a  non-adaptive  configu¬ 
ration.  This  may  stem  from  the  fact  that  the  demands  of  this 
flight  task  were  not  sufficiently  high  to  realize  the  benefits  of 
the  adaptive  integration  of  visual  and  auditory  information. 
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DISCUSSION 

While  the  multi-sensory  interfaces  were  not  found  to  be 
superior  to  the  non-adaptive  visual  display  in  terms  of  target 
designation  accuracy  (Figure  2),  total  angular  head  motion 
(Figure  6),  or  perceived  mental  workload  (Figure  4),  they  were 
associated  with  shorter  designation  times  for  ground  targets 
that  were  initially  outside  of  the  pilot’s  field  of  view  (Figure  3 
upper  panel).  It  is  important  to  point  out  that  these  were  the 
targets  that  were  most  difficult  to  detect:  they  contrasted 
poorly  with  the  terrain,  making  it  difficult  to  segregate  them 
from  the  background.  It  should  also  be  noted  that  the  time 
advantage  for  the  multi-sensory  interfaces  -  approximately 
825  ms  -  may  be  of  considerable  practical  significance,  given 
the  overwhelming  temporal  demands  that  often  confront  tacti¬ 
cal  aviators. 

It  was  surprising  that  performance  efficiency  under  the 
non-adaptive  auditory  condition  was  found  to  be  inferior  to  the 
non-adaptive  visual  condition.  One  plausible  explanation  for 
this  result  is  the  technical  limitations  of  the  spatial  audio  dis¬ 
play  (i.e.,  coarse  resolution,  non-individualized  head-related 
transfer  functions,  time  delay  in  the  system,  etc.),  a  line  of 
reasoning  that  is  consistent  with  the  data  provided  by  Bolia 
and  his  colleagues  (in  press). 

A  particularly  striking  outcome  that  results  from  the  pres¬ 
ent  investigation  is  that  all  cued  (Visual,  Auditory,  and  Audi¬ 
tory  +  Visual)  interface  conditions  were  associated  with  signifi¬ 
cantly  lower  ratings  of  overall  workload  than  the  Non-Cueing 
condition.  This  result  is  consistent  with  the  analysis  of  the 
head  motion  data  (Figures  5  and  6),  which  clearly  demon¬ 
strated  that  the  cued  interface  conditions  produced  more  effi¬ 
cient  search  strategies  (i.e.,  less  head  motion)  than  the  non- 
cued  display. 
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